Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2014 


Institutionalizing  Blended  Learning  into  Joint  Training: 
A  Case  Study  and  Ten  Recommendations 


David  Fautua,  Ph.D. 

Joint  Staff 
J7,  Joint  Training 


Sae  Schatz,  Ph.D. 

MESH  Solutions,  LLC™ 
(a  DSCI®  Company) 


Emilie  Reitz 

Alion 

Science  and  Technology 


Patricia  Bockelman, 

MESH  Solutions,  LLC™ 
(a  DSCI®  Company) 


Suffolk,  VA 


Orlando,  FL 


Suffolk,  VA 


Orlando,  FL 


david.t.fautua.civ@mail.mil  saeschatz@gmail.com 


emilie.a.reitz.ctr@mail.mil  pbockelman@mesh.dsci.com 


ABSTRACT 

In  2011,  the  Joint  Staff  J7  (Joint  Training)  directorate  initiated  the  Continuum  of  eLearning  project  in  order  to 
integrate  blended  learning  into  joint  exercises.  This  three-year  research  and  development  effort  included 
construction  of  both  the  blended  learning  instructional  materials  (e.g.,  best  practices  for  online  instructional  delivery 
[andragogy]  within  Joint  Knowledge  Online)  and  the  processes  required  to  implement  blended  learning  within  the 
existing  joint  training  enterprise.  Although  the  capacity  for  blended  learning  has  existed  for  decades,  such  large- 
scale  institutionalization  of  it  presented  unique  challenges,  which  have  previously  limited  its  use  within  the  joint 
training  community. 

Joint  Training  personnel  built  the  blended  learning  system  iteratively,  concurrently,  and  incrementally  over  the 
three-year  project.  We  also  systematically  measured  the  effectiveness  of  implemented  components.  This  paper 
presents  an  overview  of  this  process  as  a  case  study  for  others,  and  it  summarizes  the  results  of  the  empirical  testing. 
The  paper  builds  upon  two  previous  I/ITSEC  presentations,  each  of  which  detailed  separate  portions  of  the  ongoing 
project  (i.e.,  effectiveness  of  blended  online  courses  in  2012  and  integration  of  a  part-task  team-training  simulation 
in  2013).  This  paper  adds  to  those  earlier  articles  by  presenting  holistic  project  outcomes,  along  with  previously 
unpublished  data  from  the  empirical  trials.  For  instance,  some  notable  results  included  21%  higher  learning 
outcomes  (knowledge)  when  complementary  e-learning  courses  preceded  an  exercise,  and  when  an  additional  team¬ 
training  simulation  was  added  to  the  pre-exercise  preparation,  62.9%  of  participants  indicated  that  they  felt  more 
confident  performing  their  assigned  tasks  as  a  result.  Finally,  this  paper  includes  ten  recommendations  for  other 
organizations  seeking  to  formally  implement  a  blended  learning  system,  including  don’t  assume  that  trainers  know 
how  to  “blend”  from  an  instructional  perspective  and  to  truly  implement  a  robust  blended  learning  system  the 
organizational  culture  and  its  shared  narrative  about  training  must  evolve. 

ABOUT  THE  AUTHORS 

David  T.  Fautua,  Ph.D.,  is  Chief,  Individual  Training  &  Learning,  Joint  Training,  J7  Joint  Staff.  He  formerly 
served  as  U.S.  Joint  Forces  Command  Academic  Chair,  held  an  appointment  as  visiting  associate  professor  at  the 
Joint  Forces  Staff  College,  and  was  special  assistant  to  two  NATO  Supreme  Allied  Commanders.  Most  recently,  he 
served  as  the  principal  investigator  for  the  NTSA-award  winning  Border  Hunter  research  project. 

Sae  Schatz,  Ph.D.,  was  the  Chief  Scientist  of  MESH  Solutions,  LLC™  (A  DSCI®  Company)  when  this  work  was 
conducted.  Her  work  focuses  on  education  and  training  for  the  Defense  community,  particularly  the  skills  associated 
with  “cognitive  readiness.”  Dr.  Schatz  is  also  an  adjunct  instructor  with  the  UCF  Modeling  and  Simulation  program. 

Emilie  Reitz,  M.A.,  is  a  Research  Analyst  at  Alion  Science  and  Technology.  She  is  currently  supporting  the  Joint 
Fires  Division  of  Joint  Staff  J6,  Deputy  Director  for  Command  and  Control  Integration  (C2I).  In  this  capacity,  she  is 
the  data  collection  and  analytical  working  group  lead  for  Bold  Quest.  Her  research  focuses  on  integrating  joint 
capabilities  into  modeling,  simulation,  and  training,  as  a  performance  enabler. 

Patricia  Bockelman,  Ph.D.,  is  the  Chief  Scientist  at  MESH  Solutions,  LLC™  (A  DSCI®  Company).  Her 
background  includes  research  in  cognitive  science,  intelligent  behavior,  learning  and  training,  and  simulation 
environments.  Her  expertise  includes  applying  interdisciplinary  techniques  to  combine  neurophysiological, 
psychological,  and  phenomenological  metrics  to  evaluate  human  experience. 


2014  Paper  No.  14208  Page  1  of  11 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

DEC  2014 


2.  REPORT  TYPE 


3.  DATES  COVERED 


00-00-2014  to  00-00-2014 


4.  TITLE  AND  SUBTITLE 

Institutionalizing  Blended  Learning  into  Joint  Training:  A  Case  Study 
and  Ten  Recommendations 

6.  AUTHOR(S) 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 


5d.  PROJECT  NUMBER 


Conference,  1-4  Dec  2014, 


5e.  TASK  NUMBER 
5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES)  8.  PERFORMING  ORGANIZATION 

Joint  Staff, J7,  Joint  Training, Suffolk, VA, 23432  REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES)  10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

I/ITSEC  2014  -  Interservice/Industry  Training,  Simulation  and  Education  Conference,  1-4  Dec  2014, 
Orlando,  FL. 

14.  ABSTRACT 

In  2011,  the  Joint  Staff  J7  (Joint  Training)  directorate  initiated  the  Continuum  of  eLearning  project  in 
order  to  integrate  blended  learning  into  joint  exercises.  This  three-year  research  and  development  effort 
included  construction  of  both  the  blended  learning  instructional  materials  (e.g.,  best  practices  for  online 
instructional  delivery  [andragogy]  within  Joint  Knowledge  Online)  and  the  processes  required  to 
implement  blended  learning  within  the  existing  joint  training  enterprise.  Although  the  capacity  for  blended 
learning  has  existed  for  decades,  such  large-scale  institutionalization  of  it  presented  unique  challenges, 
which  have  previously  limited  its  use  within  the  joint  training  community.  Joint  Training  personnel  built 
the  blended  learning  system  iteratively,  concurrently,  and  incrementally  over  the  three-year  project.  We 
also  systematically  measured  the  effectiveness  of  implemented  components.  This  paper  presents  an 
overview  of  this  process  as  a  case  study  for  others,  and  it  summarizes  the  results  of  the  empirical  testing. 
The  paper  builds  upon  two  previous  I/ITSEC  presentations,  each  of  which  detailed  separate  portions  of  the 
ongoing  project  (i.e.,  effectiveness  of  blended  online  courses  in  2012  and  integration  of  a  part-task 
team-training  simulation  in  2013).  This  paper  adds  to  those  earlier  articles  by  presenting  holistic  project 
outcomes,  along  with  previously  unpublished  data  from  the  empirical  trials.  For  instance,  some  notable 
results  included  21%  higher  learning  outcomes  (knowledge)  when  complementary  e-learning  courses 
preceded  an  exercise,  and  when  an  additional  team-training  simulation  was  added  to  the  pre-exercise 
preparation,  62.9%  of  participants  indicated  that  they  felt  more  confident  performing  their  assigned  tasks 
as  a  result.  Finally,  this  paper  includes  ten  recommendations  for  other  organizations  seeking  to  formally 
implement  a  blended  learning  system,  including  don???t  assume  that  trainers  know  how  to  ???blend??? 
from  an  instructional  perspective  and  to  truly  implement  a  robust  blended  learning  system  the 
organizational  culture  and  its  shared  narrative  about  training  must  evolve. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF 

ABSTRACT 

OF  PAGES 

RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

it 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Inters ervic e/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2014 


Institutionalizing  Blended  Learning  into  Joint  Training: 
A  Case  Study  and  Ten  Recommendations 


David  Fautua,  Ph.D. 

Joint  Staff 
J7,  Joint  Training 


Sae  Schatz,  Ph.D. 

MESH  Solutions,  LLC™ 
(a  DSCI®  Company) 


Emilie  Reitz 

Alion 

Science  and  Technology 


Patricia  Bockelman, 

MESH  Solutions,  LLC™ 
(a  DSCI®  Company) 


Suffolk,  VA 


Orlando,  FL 


Suffolk,  VA 


Orlando,  FL 


david.t.fautua.civ@mail.mil  saeschatz@gmail.com 


emilie.a.reitz.ctr@mail.mil  pbockelman@mesh.dsci.com 


INTRODUCTION 


Teachers  and  trainers  have  relied  upon  blended  learning  for  at  least  two 
decades,  and  scholars  have  authored  (and  continue  to  write!)  thousands 
of  papers  about  it.  Yet,  despite  the  wealth  of  individual  experience  and 
reams  of  published  theory  available,  institutionalizing  high-quality 
blended  learning  processes  in  large,  dispersed  organizations  still  poses 
significant  challenges.  The  Joint  Staff  J7,  Deputy  Director  for  Joint 
Training  initiated  the  Continuum  of  eLearning  (CoL)  project  in  order  to  overcome  barriers  to  institutionalization  of 
integrating  blended  learning  into  the  joint  training  enterprise.  This  enterprise  affects  thousands  of  personnel  each 
year,  and  diverse  service  and  program  office  stakeholders  support  it.  Hence  the  CoL  project  involved  identifying 
ways  to  systematize  instructional  best  practices,  build  a  common  vision  for  blended  learning  among  the  various 
stakeholders,  formalize  organizational  processes  for  it,  and  empirically  demonstrate  the  system’s  feasibility  and 
value.  As  of  2014,  Joint  Staff  J7  (Joint  Training)  has  implemented  the  resulting  Blended  Learning-Training  System 
( BLTS ),  comprising  the  unified  blended  learning  concept,  corresponding  set  of  processes,  and  growing  repository  of 
blended  learning  materials.  We  have  demonstrated  its  viability  at  eight  combatant  command  training  events  and 
collected  data  on  its  effectiveness  at  PANAMAX  2012,  Savannah  Shield  2013,  and  Turbo  Challenge  2014.  This 
paper  summarizes  the  three-year  research  project,  data  collection  outcomes,  and  resulting  BLTS;  it  also  includes 
generalizable  lessons  learned  about  institutionalizing  blended  learning  in  complex  organizations. 

JOINT  TRAINING  EXERCISES 

Each  year  the  Joint  Staff  J7  (Joint  Training)  directorate  supports  dozens  of  large-scale  training  events  on  behalf  of 
the  combatant  commands  and  their  service  component  commands.  These  challenging,  fast-paced  exercises  include  a 
wide  range  of  training  objectives  that  must  be  accomplished  on  an  intense  schedule  and  (particularly  of  late)  on  a 
constricted  budget  (see  Figure  1).  Military  leaders,  therefore,  are  continuously  searching  for  ways  to  enrich  the 
events’  training  outcomes  and  maximize  scarce  resources. 


BLENDED  LEARNING  -  Combining 
online  and  face-to-face  instruction;  often  used 
interchangeably  with  the  term  hybrid 
learning;  see  Bonk  and  Graham  (2006)  for 
detailed  definitions  and  history. 


Classroom-based  academics  in  support 
of  PANAMAX  (2012) 

Photo:  MC2(SW)  Robert  A.  Wood  Sr. 


Personnel  simulate  a  medical  evacuation 
during  Ultimate  Caduceus  (2014) 
Photo:  Staff  Sgt.  Stephenie  Wade 


Operational  staff  use  computer 
simulation  in  Vibrant  Response  (2013) 
Photo:  Sgt.  Ken  Scar 


Figure  1.  Joint  training  exercises  generally  involve  a  combination  of  classroom-based  learning  and  hands-on  scenario- 
based  training,  including  both  computer-based  simulation  and  hands-on  field  training  (examples  shown) 
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In  2011,  Joint  Staff  J7  initiated  the  CoL  project  in  order  to  explore  integration  of  blended  learning  into  collective 
training  exercises.  This  exploration  was  motivated  by  observed  gaps  associated  with  training  events  (for  more  detail 
on  these  gaps,  see  Fautua,  Schatz,  Reitz,  &  Killilea,  2012).  In  brief,  the  gaps  included: 

(1)  “Untrained”  staff:  Up  to  40%  of  staff  members  might  miss  a  given  training  event  (e.g.,  due  to  scheduling) 

(2)  Stovepiped  training  and  education:  The  various  learning  opportunities  within  an  exercise  lacked  integration 

(3)  Service-specific  mindsets:  Some  personnel  lacked  the  knowledge  and  attitudes  needed  for  joint  operations 

(4)  Insufficient  data:  A  lack  of  targeted,  objective  assessment  of  personnel’s  incoming  subject-matter  knowledge 

(5)  Retention:  Staff  members’  knowledge  potentially  decayed  between  the  annual  (or  longer)  training  events 

To  address  these  gaps,  Joint  Training  personnel  developed  the  BLTS.  This  system  complements  the  Joint  Training 
System  (see  U.S.  Joint  Staff  doctrinal  publications  CJCS  Guide  3501  and  CJCSM  3500.03D ),  which,  similar  to  the 
well-known  ADDIE  model  of  instructional  design  (Branson  et  al.,  1975),  defines  deliberate  processes  for  designing, 
planning,  executing,  evaluating,  and  assessing  joint  training.  The  BLTS  includes  a  corresponding  set  of  processes 
for  developing,  executing,  and  continuously  improving  blended  learning  within  the  joint  training  context.  The  BLTS 
documentation  also  articulates  the  concept  of  and  rationale  for  employing  joint  blended  learning,  and  the  system 
includes  a  growing  repository  of  blended  learning  instructional  materials,  such  as  tailored  courseware,  training 
scenarios,  and  supporting  multimedia  vignettes  (see  Figure  2).  In  the  future,  we  anticipate  that  the  BLTS  concept, 
processes,  and  content  repository  will  merge  into  the  Joint  Training  System,  as  ongoing  use  of  blended  learning 
causes  this  once-novel  concept  to  become  status  quo. 


Joint  Blended  Learning  Concept  Processes  and  Documentation  Instructional  Content 

Figure  2.  The  BLTS  comprises  (1)  the  blended  learning  concept,  (2)  a  set  of  processes,  and  (3)  a  repository  of  instructional 
materials  for  supporting  blended  learning  within  the  joint  training  enterprise  ( web-based  examples  shown) 

Individual  training  bundles  produced  by  the  BLTS  are  called  Blended  Learning-Training  Packages  (BLTPs).  Joint 
Training  personnel  design  these  training  materials  so  that  they  integrate  with  a  particular  exercise  or  support  an 
emergent  commandant  command  training  request.  In  other  words,  BLTPs  align  with  the  objectives  and  schedule  of 
the  live  training  events  that  they  support,  and  each  BLTP  includes  a  combination  of  the  following  components: 

•  Online  Courses:  Tailored,  individually  paced  e-learning  courses  from  Joint  Knowledge  Online  (JKO) 

•  Part-Task  Team  Training:  Online  distributed  simulation-based  training  designed  for  small  groups 

•  Blended  Live  Training:  Interventions  and  multimedia  to  augment  live  events 

•  Feedback  Processes:  Formalized  feedback  loop  (e.g.,  metrics  reports)  to  provide  outcome  data  to  stakeholders 

•  Instructional  Design  Processes:  Administrative  and  instructional  design  guidelines  for  BLTP  development 

•  Schedule  Processes:  Adjustments  to  the  associated  exercise’s  Joint  Event  Life  Cycle  (JELC)  timeline  (a 
process  defined  in  the  Joint  Training  System)  to  accommodate  the  front-end  integration  of  blended  learning 

BLTS  DEVELOPMENT:  CASE  STUDY 

The  BLTS  was  developed  and  integrated  with  the  Joint  Training  System  over  three  years  (2011-2014).  Initially,  the 
blended  learning  concept  faced  many  hurdles,  and  the  bureaucratic  structure  of  the  joint  training  enterprise  (e.g.,  the 
compartmentalization  of  duties  and  resources)  created  barriers  to  its  implementation.  To  address  these  concerns,  the 
project  team  used  a  human-systems  integration  (HSI)  methodology  to  establish  the  system;  this  involved  iterative, 
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incremental,  and  concurrent  research  and  development  with  an  emphasis  on  human-centric  issues  (e.g.,  see  Pew, 
Mavor,  et  al.,  2007).  In  our  opinion,  this  deliberate  approach  significantly  contributed  to  the  successful  integration  of 
the  BLTS  into  the  joint  training  enterprise.  The  following  subsections  describe  the  three-year  HSI  process  as  a 
generalizable  case  study  for  other  complex  organizations  seeking  to  implement  blended  learning. 

Phase  1  (Fiscal  Year  2012) 

Like  all  investigational  efforts,  this  project  began  with  several 
months  of  team  sensemaking,  which  involved  participating 
in  frequent  discussions,  reading  articles,  developing  graphics 
and  presentations,  and  creating  a  shared  narrative.  By  “shared 
narrative,”  we  mean  an  evocative  and  easily  understood 
“elevator  pitch”  that  explained  the  need,  expected  value,  and 
planned  approach  for  building  the  system.  (See  Table  1.) 


Another  important  aspect  related  to  team  sensemaking 
involved  gap  analysis.  Although  our  team  had  firsthand 
experience  with  joint  training  and  had  already  observed  some 
gaps,  anecdotally,  we  took  steps  to  formally  document  these 
issues  and  verify  our  subjective  understanding  of  them.  This 
involved  compiling  extant  data  (e.g.,  about  the  number  of 
personnel  who  miss  an  exercise  due  to  scheduling  issues)  and  discussing  observed  gaps  with  fellow  Joint  Staff  J7 
staff  and  decision-makers.  These  meetings  helped  refine  our  understanding  (and  shared  narrative)  of  the  gaps,  and 
they  were  a  mechanism  by  which  to  involve  other  stakeholders  in  the  BLTS  development  initiative.  Such  early 
stakeholder  involvement  proved  critical  to  the  future  success  of  the  institutionalization  effort. 

As  a  secondary  part  of  the  gap  analysis,  we  interviewed  seven  military  distance  learning  subject-matter  experts 
(SMEs)  from  both  within  and  outside  of  the  Joint  Staff  J7;  they  helped  our  team  identify  possible  barriers  to  the 
implementation  of  the  BLTS  (see  Fautua  et  al.,  2012a,  2012b,  for  a  summary  of  the  responses).  For  instance,  they 
warned  us  about  trainees’  somewhat  negative  perception  of  online  learning  and  previous  issues  they  had  all 
experienced  with  poor  quality  embedded  assessments.  These  SMEs’  past  experiences  and  recommendations  helped 
our  team  create  strategies  to  mitigate  pitfalls  that  we  might  have  otherwise  been  unprepared  for. 

Concurrently,  other  members  of  our  team  developed  the  initial  high-level  requirements  for  the  BLTS.  Although  it 
seems  counterintuitive  to  design  the  initial  system  while  simultaneously  engaging  in  sensemaking  and  analysis,  this 
approach  created  efficiencies.  (Such  concurrent  development  is  considered  a  best  practice  of  HSI  for  this  reason;  see 
Pew,  Mavor,  et  al.,  2007.)  Initially,  the  high-level  requirements  only  described  the  design  of  the  online  courses  along 
with  a  plan  for  aligning  their  content  with  joint  doctrine  and  the  training  events  they  were  designed  to  support.  The 
initial  requirements  were  based  upon  the  team’s  subject-matter  knowledge  of  e-learning  and  instructional  design, 
informed  by  doctrine  and  published  best  practices  for  blended  learning.  We  also  drew  upon  the  instructional  design 
experience  from  course  designers  in  JKO  (i.e.,  the  e-learning  team  within  Joint  Staff  J7). 

The  next  task  involved  initial  prototype  development.  That  is,  the  JKO  instructional  development  team  built  a 
preliminary,  experimental  BLTP  based  upon  the  initial  requirements  and  recommendations.  At  this  early  phase  in 
the  project,  the  BLTP  consisted  of  three  online  courses  and  a  delivery  schedule  tied  to  PAN  AM  AX  2012,  the 
exercise  where  we  tested  BLTS  Version  1.0.  Gaining  approval  to  test  the  prototype  presented  some  institutional 
challenges.  Initially,  PANAMAX  stakeholders  were  concerned  about  how  the  BLTP  would  impact  the  exercise. 
Leaders  from  the  training  audience,  for  instance,  worried  about  how  much  time  their  personnel  would  spend 
completing  the  “extra”  training  and  observer/trainers  (i.e.,  “instructors”)  from  the  Joint  Staff  J7  wondered  whether 
they  would  have  additional  tasking  in  support  of  the  BLTS.  Strategic  messaging  and  involvement  of  stakeholders 
were  paramount  for  overcoming  these  concerns.  This  meant  frequently  explaining  the  “why,  how,  and  what”  of  the 
concept,  engaging  stakeholders  in  both  formal  and  informal  professional  settings,  and  actively  listening  to  their 
concerns  and  recommendations  about  the  BLTS.  To  support  our  strategic  messaging,  we  developed  a  one -page 
summary  sheet  about  the  project,  gave  presentations  at  the  scheduled  PANAMAX  planning  meetings,  and  held 
desk-side  discussions  with  an  array  of  interested  parties. 


Table  1.  Summary  of  Tasks  for  FY  2012 


Qi 

Q2 

Q3 

Q4 

Team  sensemaking 

X 

X 

Gap  analysis 

X 

X 

Requirements  development 

X 

BLTS  (V1.0)  development 

X 

Strategic  messaging 

X 

Empirical  testing 

X 

Requirements  revision 

X 

These  task  summary  tables,  provided  in  each  subsection, 
roughly  depict  the  timing  and  duration  of  major  stages  in  the 
research  and  development  process. 
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In  large  part  due  to  the  strategic  messaging  efforts,  we 
gained  permission  to  empirically  test  the  BLTP  at 
PANAMAX.  Data  collection  took  place  20  May-17 
August  2012  in  Mayport  and  Doral,  FL  and  in  Suffolk, 
VA.  U.S.  personnel  from  Tier  I  and  Tier  II  staffs 
participated  in  the  longitudinal  data  collection 
(N  =  196).  The  experimental  group  (n  =  106) 
completed  the  BLTP.  At  times  Oi  and  O2  (see 
Figure  3),  data  were  collected  on  their  immediate 
learning  outcomes  in  the  online  courses  (i.e.,  pre  and 
post-tests),  and  they  completed  surveys  about  the 
courses’  usability,  motivational  effects,  operational 
relevance,  and  ability  to  engender  a  “joint  mindset.” 
The  experimental  participants,  along  with  the  control  group  participants  ( n  =  90),  completed  knowledge  tests,  joint 
mindset  questionnaires,  and  reactions  surveys  during  the  PANAMAX  exercise  at  times  O3  and  O4  in  the  study 
diagram,  after  the  classroom  academics  and  full-scale  exercise,  respectively.  (See  Fautua  et  al.,  2012b  for  a  technical 
report  of  the  experimentation;  a  subset  of  the  most  notable  results  are  described  below.) 


Experimental  O ^ 

^Online  ^2  ^Acad 

o3 

XEX 

0, 

Control 

^Acad 

os 

Xex 

0, 

O  =  Measure  X  =  Intervention 


Figure  3.  Phase  1  Study  Design 

Oi  and  02  represent  online  testing  opportunities,  while  X0nune  signifies 
the  online  beta  courses  and  XAcad  is  academics.  O3  represents  those 
measurement  opportunities  that  occurred  just  prior  to  the  PANAMAX 
exercise,  and  O4  indicates  those  that  occurred  immediately  following  it, 
while  XEx  represents  the  exercise. 


Learning.  All  participants  completed  identical  knowledge  tests  at 
the  end  of  the  PANAMAX  exercise.  Outcomes  from  a  univariate 
between- subjects  analysis  of  covariance  (ANCOVA),  controlling 
for  rank  and  service,  showed  a  statistically  significant  difference 
between  the  scores  from  the  control  group  (ft  =  47,  M-  48.83%) 
and  experimental  group  (n-  57,  M-  59.09%),  F( 2, 48)  =  8.58, 
p  =  .001.  Personnel  in  the  experimental  condition  scored,  on 
average,  21%  better  on  the  knowledge  test  versus  participants  in 
the  control  group  (see  Figure  4).  (Note,  because  data  collection 
took  place  over  a  three-month  period  and  in  an  ecological  setting,  Figure  4.  PANAMAX  Learning  Outcomes 
attrition  occurred;  the  reduced  ft- values  reflect  this.) 


Table  2.  Meaningful  trends  in  free- text  reactions 


Reactions.  Participants  completed  several  reaction 
surveys  through  the  longitudinal  study.  For  instance, 
after  passing  the  online  courses,  experimental 
participants  responded  to  several  five -point  Likert- 
style  items,  and  they  generally  agreed  (i.e.,  4  out  of  5) 
that  the  online  courses  better  prepared  them  for 
operating  in  a  joint  and  multinational  environment 
(repeated  measures  aggregated  across  courses:  11% 
disagree ,  34%  neutral ,  and  43%  agree).  Some  of  the 
most  notable  responses,  however,  came  from  the 
survey  given  at  the  end  of  the  full  exercise,  which  included  a  section  for  detailed  free-text  responses.  Select 
items  were  worded  so  that  both  experimental  and  control  participants  could  respond  to  them.  To  analyze  the 
responses,  two  researchers  reviewed  the  data,  categorized  the  most  common  remarks,  and  tallied  the  percentage 
of  participants  who  provided  similar  feedback.  A  comparison  between  the  experimental  and  control  group 
responses  revealed  three  notable  differences.  In  particular,  82%  of  experimental  participants  indicated  that  they 
felt  prepared  for  the  event,  in  comparison  to  only  18%  of  the  control  group  (see  Table  2  for  more  details). 


Participant  Remarks 

Control 

Exp. 

Online  pre-training  would  have  been 
(or  was)  beneficial 

68% 

92% 

Participant  felt  prepared  for 
academics  and/or  exercise 

18% 

82% 

Participant  felt  rushed  to  complete 
training  requirements  prior  to  exercise 

68% 

71% 

To  summarize  the  key  points  from  data  collection  at  PANAMAX,  the  training  audience  reported  lukewarm 
subjective  reactions  about  the  online  courses  designed  to  coincide  with  the  exercise;  however,  they  showed 
higher  levels  of  knowledge  at  the  end  of  the  live  training  exercise  (as  compared  to  the  control  group).  Finally, 
when  asked  to  reflect  at  the  end  of  the  live  training  exercise,  the  experimental  participants  were  substantially 
more  likely  to  report,  retrospectively,  that  they  were  well  prepared  coming  into  the  event. 


The  practical  experience  gained  by  the  designing  and  executing  the  PANAMAX  prototype,  along  with  the  formal 
data  collected,  gave  the  project  team  new  insights.  We  revised  and  expanded  the  BLTS  requirements,  and  offered 
research-informed  recommendations  for  practical  next  steps  for  its  integration  into  the  joint  training  context. 
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Phase  2  (Fiscal  Year  2013) 


Immediately  after  PANAMAX,  Joint  Training  personnel 
began  making  improvements  to  the  learning  management 
system  (LMS),  in  response  to  recommendations  uncovered 
during  the  empirical  trial.  Joint  Staff  J7  personnel  also  began 
working  on  BLTS  (V2.0)  development,  which  included  the 
addition  of  a  ((data  dashboard ”  for  reviewing  interim  output 
data  and  a  part-task  team-training  simulation  for  distributed 
pre-event  teamwork  training.  (See  Table  3.) 

Data  Dashboard.  The  data  dashboard  is  a  report  summarizing  training  audience  data  from  the  online  courses. 
During  the  PANAMAX  exercise,  the  trainers  and  training  audience  commanders  only  received  outcome  data 
from  the  BLTS  post-exercise.  However,  they  needed  to  review  that  data  before  the  event,  in  order  to  tailor  their 
instructional  approach  to  fit  participants’  demographics  and  identified  knowledge  gaps.  A  rudimentary  version 
of  the  data  dashboard  was  piloted  during  SOUTHCOM’s  Integrated  Advance  2012  exercise,  just  a  few  months 
after  PANAMAX  2012.  While  not  a  formal  study,  comments  in  the  event’s  after-action  report  suggest  it  was 
positively  received.  The  U.S.  Army  South  Chief  of  Training  noted  that  incorporation  of  this  interim  feedback 
loop  was  a  potential  game-changer  for  the  design  of  major  exercises  because  (once  mature)  it  would  allow 
trainers  to  target  and  optimize  the  experience  for  each  unique  training  audience,  where  previously  mechanisms 
for  this  sort  of  tailoring  did  not  readily  exist  (C.  Unrath,  personal  communication,  March  21,  2013). 

Team-Training  Simulation.  The  second  major  addition  to  the  BLTS  included  a  distributed  part-task  team¬ 
training  simulation.  From  an  instructional  design  perspective,  trainees  (in  groups  of  about  10-40)  are  intended 
to  complete  scenarios  in  the  part-task  simulation  after  they  take  the  individual  online  courses  but  before  they 
attend  the  full-scale  collective  exercise.  Presently,  this  simulation-based  training  is  delivered  via  the  Small 
Group  Scenario  Trainer  (SGST)  software  and  served  through  the  Joint  Knowledge  Online  backbone.  (See  also 
Reitz  et  al.,  2013,  which  summarizes  the  part-task  team-training  additions  to  the  BLTS.) 

While  the  developers  worked  on  the  BLTS  enhancements  and  expansions,  other  team  members  conducted  a  series  of 
stakeholder  interviews  intended  to  gauge  the  level  of  buy-in  for  the  BLTS  and  to  uncover  additional 
recommendations  for  it.  Twenty-three  (N  =  23)  Joint  Staff  J7  personnel  participated,  including  nine 
observer/trainers,  five  desk  officers  and/or  exercise  planners,  and  five  personnel  from  JKO.  During  the  interviews, 
stakeholders  expressed  strong  support  for  the  blended  learning  concept — a  marked  change  from  the  previous  year, 
when  many  viewed  blended  learning  with  deep  reservations.  However,  the  interviewees  recognized  that  the  need  to 
invest  even  greater  effort  into  socializing  the  blended  learning  concept,  both  internally  within  the  directorate  as  well 
as  with  stakeholders  from  the  combatant  commands.  Interviewees  also  raised  questions  about  the  administrative 
processes  associated  with  the  BLTS,  offering  suggestions  on  how  to  streamline  the  creation,  verification,  and 
maintenance  procedures,  and  suggesting  ways  to  better  align  the  online  courses  and  team-training  scenarios  with 
their  corresponding  live  exercises.  Finally,  a  number  of  interviewees  recommended  that  the  blended  learning  “team” 
needed  to  work  more  closely  with  the  deployable  training  team  (DTT)  assigned  to  an  event,  so  that  the  online 
components  received  sufficient  attention  and  resourcing.  This  included  tighter  incorporation  with  existing  JELC 
(i.e.,  the  formal  processes  outlined  in  the  Joint  Training  System  for  planning  and  executing  individual  exercises). 
Based  upon  the  interviews,  the  researchers  derived  new  recommendations  for  the  BLTS,  and  from  our  anecdotal 
observations,  conducting  these  interviews  helped  spread  information  about  and  encourage  greater  buy-in  for  the 
system.  (These  interviews  are  outlined  in  a  technical  report;  see  Schatz,  Killilea,  &  Reitz,  2013.) 

At  the  end  of  fiscal  year  2013,  the  research  team  prepared  for  another  empirical  testing  opportunity,  this  time  with 
an  emphasis  on  the  newly  integrated  part-task  team-training  component  of  the  BLTS.  Joint  Training  developers 
created  a  BLTP  aligned  with  the  Judicious  Response  2013  exercise;  however,  instability  surfaced  in  the  combatant 
command’s  area  of  responsibility,  and  Judicious  Response  was  cancelled.  As  a  result,  the  research  team  could  only 
execute  a  small-scale  pilot  of  the  BLTP,  which  was  given  its  own  exercise  name  of  Savannah  Shield. 
Serendipitously,  the  limited-scale  pilot  test  uncovered  several  flaws  in  the  administrative  processes  associated  with 
the  execution  of  the  part-task  trainer;  the  following  paragraphs  outline  some  examples  of  these  issues. 


Table  3.  Summary  of  Tasks  for  FY  2013 


LMS  improvements 


BLTS  (V2.0)  development 
Stakeholder  interviews 
Empirical  testing  (pilot) 


Requirements  revision 
Strategic  messaging 
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The  Savannah  Shield  BLTP  began  with  five  e-learning  courses:  two  lower-level  fundamentals  modules  and 
three  courses  designed  in  support  of  Savannah  Shield.  All  participants  were  assigned  to  take  the  courses; 
however,  many  participants  failed  to  complete  them.  Later,  the  lack  of  consistent,  foundational  knowledge 
among  the  pilot-test  participants  proved  to  be  an  issue.  Fourteen  (N  =  14)  participants,  co-located  in  a  single 
computer  lab,  completed  some  parts  of  the  part-task  team- training  pilot,  delivered  via  the  SGST  software. 
However,  during  this  one-day  training  event,  those  14  staff  members  logged  in  and  out  of  the  system, 
periodically  attending  to  other  events  during  the  training,  and  some  assigned  participants  chose  not  to  attend  at 
all.  Further,  the  trainees  did  not  gather  as  a  group  until  the  event  began,  which  limited  the  amount  time  available 
for  system  familiarization  (“buttonology”  training),  discussion  of  exercise  goals,  and  any  opportunity  to  talk 
about  individual  roles  or  responsibilities.  As  a  result,  the  opportunity  to  train  an  intact  staff  team  was  not 
optimized  and  the  disrupted  training  environment  diminished  the  impact  of  the  learning  opportunity  for  the 
participants. 

The  research  team  had  prepared  a  robust  data  collection 
approach;  however,  sub-optimal  participation  in  the  training 
limited  the  utility  of  these  measures,  and  in  the  end,  the  most 
interpretable  results  came  from  the  reaction  surveys.  Only  nine 
Savannah  Shield  participants  completed  a  30-item  reaction 
survey  at  the  end  of  the  simulation-based  training  (n  =  9;  the 
other  five  trainees  chose  not  to  participate  in  the  research);  in 
addition,  nineteen  observer/trainers  (n=  19)  completed  the  same 
questionnaire  during  a  pre-event  run-through  of  the  scenario,  a 
few  days  prior.  Table  4  summarizes  the  results  of  both  groups’ 
surveys.  All  items  were  recoded  so  that  strongly  agree  (i.e.,  5 
out  of  5)  is  the  most  positive  response  while  strongly  disagree  (i.e.,  1  out  of  5)  represents  the  most  negative.  No 
meaningful  trends  in  individual  subscales  were  noted;  thus,  the  percentages  in  Table  4  represent  an  aggregation 
across  all  30  items  on  the  reaction  questionnaires. 

In  general,  both  the  observer/trainers  and  training  audience  saw  value  in  completing  a  part-task  team  training 
scenario  prior  to  a  larger-scale  collective  exercise.  Approximately  half  of  the  participants  agreed  (4  out  of  5) 
that  the  team-training  component  was  beneficial  and  a  good  complement  to  the  other  Savannah  Shield  BLTP 
training  elements  (i.e.,  the  online  courses  and  planned  collective  exercise).  However,  the  open-ended  responses 
from  the  reaction  surveys,  along  with  the  behavioral  observation  checklists  submitted  by  the  observer/trainers 
and  the  event’s  after-action  report,  all  strongly  suggested  that  refined  business  processes  were  required.  In  sum, 
the  training  components  seemed  effective  but  the  processes  for  delivering  them  needed  revision. 

Despite — or,  more  precisely,  because  of — the  challenges  encountered  during  the  Savannah  Shield  pilot  test,  the 
project  team  developed  a  variety  of  recommendations  for  new  business  processes.  These  lessons  learned  also 
informed  ongoing  requirements  development  and  our  continuous  strategic  messaging  about  the  system. 

Phase  3  (Fiscal  Year  2014) 

In  FY  2014,  the  blended  learning  team  continued  to  make 
incremental  improvements  to  the  BLTS — in  particular,  to 
its  documented  logistical  processes.  We  shared  lessons 
learned  from  the  Savannah  Shield  pilot  test  with  other 
learning  officers  and  took  steps  to  mitigate  process  gaps  in 
future  iterations.  This  benefitted  the  Joint  Force  Headquarters 
National  Capital  Region  (JFHQ-NCR),  when  the  blended 
learning  team  supported  their  out-of-cycle  training  request  in 
December  2013,  and  the  new  processes  substantially  improved  delivery  of  the  BLTS  during  the  Turbo  Challenge 
and  Ultimate  Caduceus  exercises.  (See  Table  5.) 

In  April  2014,  we  empirically  tested  the  BLTS  V3.0  during  Turbo  Challenge  and  Ultimate  Caduceus,  two 
concurrently  executed  U.S.  Transportation  Command  (TRANSCOM)  exercises.  Turbo  Challenge  emphasized 
transportation-related  actions,  and  Ultimate  Caduceus  focused  on  associated  patient-movement  objectives.  Both 


Table  5.  Summary  of  Tasks  for  FY  2014 


Ql 

Q2 

Q3 

Q4 

Incremental  improvements 

X 

X 

BLTS  (V3.0)  development 

X 

X 

Empirical  testing 


Future  planning 


Strategic  messaging 


X  X 


Table  4.  Savannah  Shield  Reactions 


Observer/ 

Trainers 

Training 

Audience 

Strongly  disagree  (1) 

5% 

0% 

Disagree  (2) 

18% 

13% 

Neutral  (3) 

26% 

32% 

Agree  (4) 

44% 

47% 

Strongly  Agree  (5) 

6% 

8% 

1  =  Most  Negative  -to-  5  =  Most  Positive 
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exercises  supported  U.S.  Northern  Command’s  Ardent  Sentry  exercise.  The  BLTPs  developed  for  Turbo  Challenge 
and  Ultimate  Caduceus  included  different  content,  but  they  shared  identical  structures,  development  processes,  and 
execution  timelines.  Both  BLTPs  consisted  of  two  online  courses  and  a  part-task  team- training  scenario,  aligned 
closely  to  narrative  and  objectives  of  the  exercises.  The  BLTPs  also  included  two  scheduled  instances  where  the 
observer/trainers  received  interim  metrics  reports  on  trainees’  demographics  and  online  performance.  Refer  to 
Figure  5  for  the  delivery  timeline  and  Figure  6  for  the  study  design. 


2  Jan -26  March  14 


Two  Online  Courses 


Two  Online  Courses 


[]  =  Blended  Learning-Training  Element 


Experimental 

Intervention 


Turbo  Challenge 


Ultimate  Caduceus 


27  Mar -3  April  14 


Figure  5.  Timeline  of  Delivery  for  the  Turbo  Challenge  and  Ultimate  Caduceus  BLTPs 


More  than  200  staff  members  participated  in  Turbo  Challenge  and  Ultimate  Caduceus,  and  TRANSCOM 
leadership  assigned  e-learning  courses  to  a  majority  of  these  personnel  (depending  upon  their  demographics). 
The  training  audience  could  access  the  e-learning  courses  up  to  three  months  prior  to  the  event,  and  92 
participants  completed  at  least  one  course.  Staff  members  who  participated  in  our  research  were  assigned  to 
complete  two  courses.  That  is,  both  experimental  and  control  participants  completed  the  e-learning  courses. 

Intervention  and  Data  Collection.  Two- weeks 
before  the  exercises,  experimental  participants 
from  Turbo  Challenge  and  Ultimate  Caduceus 
completed  separate  team  training  scenarios 
( n  =  27  and  n-  12,  respectively).  The  Turbo 
Challenge  scenario  focused  on  mission 
analysis  in  the  context  of  civilian  disaster 
response,  and  the  Ultimate  Caduceus  scenario 
involved  patient  movement  training  in  the 
same  context.  The  live  training  exercises  were 
held  from  27  March-3  April  2014.  Due  to  the 
typical  attrition  expected  in  any  applied 
research  study,  data  from  only  sixty  personnel  (N  =  60;  experimental  group  n  =  31;  control  group  n  =  29)  were 
collected  during  the  exercises  at  time  O4.  The  set  of  experimental  apparatus  included  surveys  used  in  Savannah 
Shield,  refined  based  upon  feedback  from  that  event,  and  three  new  knowledge  tests. 


Experimental 

Di  ^Online 

o3 

^■Acad 

^Team  O3  Xex 

04 

Control 

Dl  ^Online 

o3 

^■Acad 

Xe* 

04 

O  =  Measure 

X  =  Intervention 

Figure  6.  Phase  3  Study  Design 

Oi  and  O2  represent  online  testing,  and  X0nune  signifies  the  online  courses. 
XAcad  is  academics.  XTeam  denotes  the  experimental  intervention  ( part-task 
team  trainer ),  and  03  represents  the  feedback  surveys  after  it.  XEx  indicates 
the  exercises,  and  O4  denotes  the  data  collection  at  their  conclusion. 


Learning.  Participants  in  the  experimental  condition  correctly 
answered  a  greater  percentage  of  questions  on  the  three  subject- 
matter  knowledge  tests  delivered  at  the  end  of  the  exercise 
(time  O4 ).  However,  as  anticipated  (in  part  because  all  groups 
took  the  online  courses)  the  differences  in  knowledge  levels 
between  groups  were  small,  and  with  the  small  sample  size, 
they  did  not  reach  statistical  significance  (see  Figure  7). 

Reactions.  Participant  responses  to  Likert-scale  statements 

were  collected.  For  instance,  after  completing  the  team-training  _  _  „  .  ^  A  0 

.  ,  .  .  11  Figure  7.  TRANSCOM  Knowledge  Test  Scores 

scenario  (time  XTeam),  experimental  participants  were  asked  to 

agree  or  disagree  with  the  statement,  “I  am  more  confident  on  performing  the  assigned  task  than  I  was  before 


■  Experimental  Control 


Topic-1  Topic-2  Topic-3 
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training;”  68%  of  participants  indicated  that  they  agreed  or  strongly  agreed.  Further,  participants  indicated  that 
the  training  system  should  be  used  to  train  service  members  (26%  agreed ,  52%  strongly  agreed).  Later,  at  the 
end  of  the  exercise  (time  O5)  both  the  experimental  and  control  groups  were  asked  to  retrospectively  rate  their 
level  of  preparedness,  coming  into  the  event.  While  both  groups  agreed  that  they  felt  prepared  for  the  exercise 
(experimental:  71%  and  control:  62%),  the  participants  who  completed  the  part-task  training  were  more  likely 
to  strongly  agree  than  the  control  group  that  they  felt  prepared  (experimental:  13%  vs  control:  10%). 
Experimental  participants  were  also  much  less  likely  to  have  a  neutral  opinion  about  their  pre-exercise 
preparation  (experimental:  6%  vs  control:  28%).  That  is,  experimental  responses  skewed  slightly  more  positive. 

Performance.  In  addition  to  the  experimental  team,  observer/trainers  and  other  exercise  SMEs  were  enlisted  to 
collect  observational  data  about  the  applied  performance  of  personnel  in  the  live  exercise.  Though  anecdotal, 
this  input  helped  uncover  some  impacts  of  the  part-task  team  training.  For  instance,  a  member  of  the  Turbo 
Challenge  joint  planning  team  (identified  as  an  experimental  participant,  ex  post  facto)  performed  notably  well 
on  a  mission  analysis  task  but  openly  remarked  on  his  lack  of  confidence.  When  asked  why  he  felt  uncertain 
about  his  quality  performance,  he  revealed  that  he  had  not  participated  in  a  mission  analysis  effort  before  and 
his  only  experience  with  it  was  from  the  blended  learning  courses  and  part-task  trainer.  He  attributed  his 
successful  performance  to  the  part-task  pre-training  simulation. 

To  summarize  the  TRANSCOM  outcomes,  the  quantitative  data  showed  a  slight  and  generally  not  statistically 
significant  benefit  to  the  inclusion  of  the  part-task  team  trainer.  However,  the  qualitative  data  revealed 
advantages  that  were  observable  by  the  observer/trainers  on  topics  outside  of  the  scope  of  the  quantitative  tools. 

Successful  delivery  of  the  comprehensive  BLTP  at  TRANSCOM  demonstrated  that  the  BLTS  has  become  a  fully 
operational  capability  that  can  now  routinely  support  events  across  the  joint  training  enterprise.  Further,  with  this 
new  capability,  Joint  Staff  J7  can  consider  future  enhancements  to  the  Joint  Training  System  that  the  system  could 
not  logistically  support  before.  (See  Fautua,  Schatz,  et  al.,  2014  for  a  description  of  two  possible  future 
enhancements:  supporting  multi-lifecycle  event  planning  and  enabling  training  for  mission  command  mindsets 
across  echelons.)  Finally,  as  before,  throughout  FY  2014  our  team  continued  to  emphasize  the  importance  of 
strategic  messaging — including  this  paper — to  create  a  shared  vision  among  stakeholders,  exchange  data  and 
communicate  ongoing  lessons  learned,  and  receive  feedback  and  suggestions  on  our  ideas  and  methods. 

TEN  RECOMMENDATIONS 

Although  the  capacity  for  blended  learning  has  existed  for  years,  large-scale  institutionalization  of  it  presented 
unique  challenges  for  the  joint  training  enterprise.  Using  an  HSI  research  and  development  approach,  we  overcame 
these  barriers  and  successfully  created  the  concept,  processes,  and  initial  instructional  repository  for  joint  blended 
learning.  Through  this  effort,  we  uncovered  countless  lessons  learned.  Below  are  ten  high-level  recommendations 
that  we  felt  others  might  benefit  from.  (This  set  of  recommendations  is  offered  as  a  supplement  to  the  numerous  best 
practices  already  documented  throughout  the  literature,  e.g.,  Bersin,  2004.) 

(1)  Provide  the  “why.”  One  of  the  main  principles  of  adult  learning  theory  (andragogy)  holds  that  adults  are  goal- 
oriented;  adults  want  to  understand  the  purpose  and  value  of  a  given  learning  intervention.  In  a  blended  learning 
context,  stakeholders  across  all  organizational  levels  (from  individual  trainees  to  their  topmost  learning  officers) 
need  to  understand  how  the  blended  learning  content  relates  to  other  training  elements,  how  it  directly  supports  their 
missions,  and  how  it  adds  value  to  the  training  continuum.  For  the  training  audience,  emphasize  What's  In  It  For  Me 
(WIIFM)  messages  early.  For  training  developers  and  administrators,  explicitly  connect  the  dots  between  the 
blended  learning  components  and  the  larger  training  and  operational  goals.  Revisit  these  connections  frequently 
during  the  project’s  design,  development,  and  feedback  phases. 

(2)  Get  compliance  and  commitment  from  trainees  and  their  leaders.  We  realized  a  hard-won  lesson  that 
compliance  does  not  necessarily  yield  commitment.  For  example,  in  the  Savannah  Shield  pilot,  the  command’s 
leadership  requested  the  training  but  did  not  make  an  overt  display  of  their  commitment  to  it  (e.g.,  by  following-up 
with  personnel  who  did  not  complete  the  assigned  online  courses).  Plus,  because  the  WIIFM  was  not  clear  to  them, 
the  training  audience  generally  complied  by  attending  the  team  training  but  found  it  difficult  to  commit  to  it.  They 
were  not  fully  engaged  or  thoughtful,  and  as  a  result,  gained  less  value  from  the  experience. 
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(3)  Design  blended  learning  content  so  that  it  also  enables  sustainment  training.  Each  new  BLTP  aligns  with  a 
specific  live  training  event,  but  BLTPs  are  also  designed  so  that  they  can  serve  as  small-scale  standalone  training. 
That  is,  the  commands  can  readily  reuse  their  tailored  courses,  part-task  scenarios,  and  other  media  elements  to 
support  periodic  training,  such  as  on-boarding  new  staff  members  coming  into  their  agencies  or  maintaining  their 
personnel’s  knowledge  and  skills  between  larger- scale  training  events. 

(4)  Train  to  the  staff — not  to  the  exercise.  In  large  organizations,  training  developers  may  feel  tempted  to  focus  on 
the  goals  and  logistics  of  scheduled  training  events,  viewing  these  events  as  ends  unto  themselves.  The  risk  is  that 
each  event  begins  to  seem  like  a  “one-shot”  and  that  training  outcomes  are  “complete”  after  it.  When  approached  in 
this  manner,  linkages  between  training  opportunities  are  lost  and  system-level  improvements  (i.e.,  double-loop 
learning)  cannot  readily  occur.  Instead  of  focusing  on  the  major  training  events  as  outcomes,  they  should  be  viewed 
as  tools  in  a  toolbox  of  variously  sized  training  interventions.  For  the  joint  training  community,  the  BLTS  supports 
this  outlook  by  providing  new  mechanisms  for  connecting  the  different  training  events,  collecting  data  to  help 
inform  the  design  of  future  training,  and  encouraging  stakeholders  to  view  the  various  training  interventions  as  parts 
of  an  interconnected  whole. 

(5)  Embed  diagnostic  measures  that  support  the  adaptation  of  future  training.  Clearly,  the  summative 
assessments  in  any  instructional  setting  should  be  diagnostic  and  support  the  delivery  of  feedback  or  remediation  to 
learners.  In  blended  learning,  there  is  an  additional  burden  to  include  diagnostic  assessments  that  inform  the 
blending  process  across  linked  training  activities.  For  instance,  in  the  BETS  e-learning,  we  included  open-ended 
free-response  questions.  Even  though  the  online  learning  management  system  cannot  “grade”  these  responses,  the 
observer/trainers  can  review  them  prior  to  an  academics  or  exercise  event  and  gain  a  greater  understanding  of  the 
training  audience’s  mindset  than  they  could  by  simply  reviewing  the  aggregated  multiple-choice  scores. 

(6)  Schedule  opportunities  for  remediation.  Within  the  BLTS,  blended  learning  elements  are  delivered  over  the 
course  of  several  weeks,  and  the  system  includes  processes  that  enable  later  training  elements  to  be  adapted  based 
upon  results  of  the  earlier  ones.  However,  the  BLTS  originally  lacked  processes  for  inserting  ad-hoc  remediation 
activities  into  the  delivery  of  a  given  BLTP.  For  instance,  the  trainers  noticed  some  gaps  in  personnel’s  knowledge 
during  the  Turbo  Challenge  part-task  simulation,  and  their  leadership  discussed  the  issues  with  their  team  during  the 
after-action  review.  However,  we  had  not  built-in  administrative  methods  for  providing  additional  remediation  in 
response  to  those  issues.  As  a  result,  some  of  the  same  gaps  were  observed  during  the  corresponding  live  exercise. 

(7)  Do  not  assume  that  trainers  know  how  to  “blend”  from  an  instructional  perspective.  Provide  guidelines  and 
suggestions  to  trainers  on  how  to  adapt  the  delivery  of  later  training  elements  in  response  to  the  results  from  earlier 
ones.  Even  the  best  instructors  may  struggle  to  meaningfully  link  and  dynamically  adjust  one  learning  event  in 
response  to  another,  and  military  personnel  may  not  have  received  formal  training  on  these  practices.  Help  trainers 
by  exposing  them  to  general  instructional  methods  for  blending,  and  when  providing  them  with  interim  data  also 
include  recommendations  for  specific  actions  to  take  in  response  to  it.  For  example,  we  found  that  simply  giving 
observer/trainers  access  to  the  data  dashboard  did  not  enable  them  to  effectively  modify  their  delivery  of  academics; 
the  dashboard  needed  to  also  include  actionable  steps  for  adjusting  the  academics  based  upon  the  data. 

(8)  Use  blended  learning  opportunities  to  also  collect  system-wide  feedback.  Because  blended  learning 
incorporates  computer-based  training,  it  provides  an  easy  mechanism  through  which  to  collect  feedback  from 
trainees.  This  feedback  might  include  data  about  training  effectiveness,  which  helps  shape  the  design  of  immediately 
connected  or  even  future  training  events,  or  the  feedback  might  include  other  reactions  about  the  training  system. 
Incorporate  intentional  processes  into  the  blended  learning  system  to  facilitate  this  sort  of  double -loop  learning  and 
continuous  system  improvement.  (This  principle  also  closely  supports  Recommendation  #4,  above.) 

(9)  When  building  a  blended  learning  system,  emphasize  human-system  integration.  When  institutionalizing 
blended  learning  at  a  large  scale,  use  HSI  best  practices  including  concurrent,  iterative,  and  incremental  project 
management  methods  and  strong  emphasis  on  human-centric  design.  Include  stakeholders  frequently,  and  spend 
energy  on  strategic  messaging.  Finally,  embrace  interim  “failures”  (like  Savannah  Shield);  temporary  problems 
encountered  during  system  implementation  help  designers  resolve  possible  pitfalls  in  a  controlled,  smaller  scale 
setting.  Negative  results  yield  some  of  the  best  opportunities  for  further  exploration  and  growth  of  a  capability. 
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(10)  Change  the  culture.  Effectively  institutionalizing  blended  learning  involves  more  than  simply  attaching  e- 
learning  courses  to  a  live  training  event.  Producing  good  quality  blended  learning  requires  development  of  new, 
“silo-bridging”  business  processes  and  an  organizational  culture  shift  with  regard  to  the  way  stakeholders  view 
training  ( a  la  some  of  the  preceding  recommendations).  In  large-scale  organizations,  a  multiplicity  of  stakeholders 
will  interact  with  the  blended  learning  system,  and  due  to  interdependencies  among  blended  learning  elements, 
misunderstandings  or  disengagement  from  a  given  stakeholder  group  could  damage  the  entire  system.  However, 
when  stakeholders  share  a  common  vision  and  appreciation  for  blended  learning,  they  can  collectively  produce 
powerful  training  experiences  that  efficiently  hone  and  reinforce  their  staffs  knowledge,  skills,  and  attitudes,  while 
simultaneously  supporting  double-loop  learning  and  future  training  innovations. 

CONCLUSION 

In  summary,  the  joint  training  BLTS  required  three  years  to  institutionalize  as  a  value-added  capability  within  the 
Joint  Training  System,  and  we  anticipate  that  the  system  (to  include  its  concept,  processes,  and  content  repository) 
will  continue  to  mature  in  the  years  to  come.  Through  this  process,  we  uncovered  many  recommendations  and  trust 
that  these  lessons  will  help  other  agencies  as  they  develop  or  enhance  their  own  blended  learning  methods. 
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